[00:01.550 --> 00:07.790]  Welcome. Today I'm going to tell you a story about things that I learned about incident response
[00:07.790 --> 00:14.890]  by being a firefighter. Let's get to it. Here's the agenda, what we're going to cover.
[00:15.070 --> 00:19.470]  I'm going to start with a little introduction about myself, tell you a little bit about my
[00:19.470 --> 00:25.250]  background in IR, and I'm going to tell you a little bit about myself as a firefighter.
[00:25.350 --> 00:30.570]  Then we'll delve into what IR is. For those of you who don't know what IR is,
[00:30.730 --> 00:34.230]  and then we'll talk a little bit about firefighting, because even though you might
[00:34.230 --> 00:39.990]  know what IR is, you might not be familiar with firefighting beyond, obviously, putting out the
[00:39.990 --> 00:46.570]  fire. Then we'll do some comparisons between IR and firefighting. Definitely some similarities
[00:47.210 --> 00:52.630]  and some differences, of course. Then I'll describe some challenges and opportunities,
[00:52.630 --> 00:57.490]  the things that I learned from being a firefighter that I then brought back to IR.
[00:58.210 --> 01:04.710]  I'll summarize and then end with some final thoughts. Let's get going.
[01:05.590 --> 01:11.720]  All righty. So, for those of you who are not familiar with me, my name is Catherine Allman.
[01:12.070 --> 01:18.510]  I go by Investigator Chih on the internet, in the Twitter-verse, and I've been with the
[01:18.510 --> 01:24.550]  University of Buffalo for over 20 years. I'm on staff with B-Sides Rochester and a cyber camp
[01:24.550 --> 01:29.190]  that we hold. I volunteer with some staff. I've spoken at a bunch of conferences. I have some
[01:29.190 --> 01:36.170]  certifications and a couple degrees. Even those of you who know me from the IT realm may not be
[01:36.170 --> 01:41.790]  familiar with this information about me as a firefighter. I've been in the fire service since
[01:41.790 --> 01:48.410]  about 1997, and since then I've become a state fire instructor with New York State. I've done
[01:48.410 --> 01:53.790]  some credits, college credits, on firefighting. I was a certified fire investigator at both levels
[01:53.790 --> 02:00.030]  in New York State, and I was also a member of the J-Fire team. So, the fire thing is something I've
[02:00.030 --> 02:08.110]  been doing a long time. Just in case you're still not convinced, this is me at the scene of a fire,
[02:08.110 --> 02:12.770]  and this is part of the story that I'm going to tell you. I'm holding what's called a thermal
[02:12.770 --> 02:18.490]  camera. When you look into a thermal camera and you point it at something, what it's going to show
[02:18.490 --> 02:25.030]  is a difference in temperature. So, things that are darker are usually cooler in temperature,
[02:25.030 --> 02:30.490]  and things that are warmer are going to be are going to show up as lighter in color.
[02:31.050 --> 02:35.730]  We'll see why that matters later. And just in case you weren't sure,
[02:35.730 --> 02:39.310]  I'm the one with the camera, and the person to my right is our captain.
[02:40.390 --> 02:47.070]  So, now let's delve into what incident response is. I like this definition of incident response
[02:47.070 --> 02:53.070]  because it focuses on the technical components that you're going to need to analyze and contain
[02:53.070 --> 02:58.730]  an incident. We'll see in a bit why that's a little different from incident handling.
[02:58.730 --> 03:02.450]  So, you're going to be the boots on the floor person. The person who's actually focused
[03:02.450 --> 03:07.570]  on dealing with this incident directly. You're going to address network events.
[03:07.730 --> 03:12.790]  You might deal with them in a reactive or a proactive way, depending on your organization.
[03:12.790 --> 03:19.630]  And an event can be anything that's going to affect the CIA triangle. Your confidentiality,
[03:19.630 --> 03:26.710]  integrity, and availability of information. The goals of incident response are going to
[03:26.710 --> 03:32.330]  be to make sure you know when there is, in fact, an incident or, you know, some kind of
[03:32.330 --> 03:37.230]  thing going on. You're going to stop that attacker. You're going to minimize the damage
[03:37.230 --> 03:42.730]  that is caused. And ultimately, you want to prevent it from ever happening again
[03:42.730 --> 03:49.370]  or something similar from happening. These are your basic phases of incident response.
[03:49.370 --> 03:55.070]  Preparation, identification, containment, eradication, recovery, and lessons learned.
[03:55.310 --> 04:02.030]  Let's break these down a little bit. Preparation. So, during the preparation phase,
[04:02.030 --> 04:08.430]  we're going to document, document, document. You want to document both your network itself.
[04:08.450 --> 04:13.250]  You might want to document the particular machines in question. One of the most important
[04:13.250 --> 04:17.670]  things you'll want to document is where your important data is, right? Where are the keys
[04:17.670 --> 04:23.290]  to the kingdom? That's critical. You might do some tool building. Perhaps to help you
[04:23.290 --> 04:28.790]  respond better to an incident. Or perhaps to help you know when something is happening that
[04:28.790 --> 04:35.290]  shouldn't be. Then the next step is going to be awareness of the attacks. And ultimately,
[04:35.290 --> 04:41.070]  this could be something as simple as an automated ticket system that tells you that there's a
[04:41.070 --> 04:47.310]  problem. Could be from a phone call. Your containment phase, you want to stop whatever
[04:47.310 --> 04:53.570]  is happening. So, that might be patching a system, blocking some sort of C2 connection,
[04:53.570 --> 04:59.730]  maybe pulling the power out of the back of a machine. Depends. Then during the eradication
[04:59.730 --> 05:04.110]  phase, you're ultimately going to remediate. You're going to remove compromised hosts from
[05:04.110 --> 05:09.750]  your network. Whatever it is you have to do to get them to get the bad guys gone.
[05:10.490 --> 05:15.550]  During the recovery phase, you need to restore those business functions. Because, of course,
[05:15.550 --> 05:21.070]  that is the whole point of incident response initially, is to get up and running again.
[05:21.070 --> 05:26.130]  Then you want to learn from exactly what happened so that the same thing doesn't happen again.
[05:27.650 --> 05:32.790]  Here we have incident handling. You'll notice it's a little different. It focuses on the other
[05:32.790 --> 05:38.190]  things that are important in an incident. The logistics, communication, the coordination,
[05:38.190 --> 05:43.950]  planning, all of these other functions. And ideally, you want these two things to be separate.
[05:44.570 --> 05:50.070]  Now, I understand in smaller organizations, you might be the only person doing this job.
[05:50.070 --> 05:54.870]  You might have to do both. But if you're the person whose boots on the ground and you're
[05:54.870 --> 06:02.250]  working on containing an incident or eradicating the attacker, you don't also want to have to be
[06:02.250 --> 06:07.210]  the person who's communicating out every five minutes what's going on. So, if you have somebody
[06:07.210 --> 06:14.570]  else doing that, it's a whole lot easier to focus on what's at hand. Now let's move to firefighting.
[06:14.570 --> 06:20.290]  What is firefighting? Well, it's the obvious. We're going to put the wet stuff on the red stuff,
[06:20.290 --> 06:26.490]  as they say. We're going to prevent the spread and extinguish significant unwanted fires. So,
[06:26.490 --> 06:31.870]  you know, not your backyard barbecue unless it's out of control. And we'll do that for buildings
[06:31.870 --> 06:38.930]  and vehicles, woodlands, that sort of thing. Our goals, as you might expect, are to protect the
[06:38.930 --> 06:45.290]  health and safety of ourselves and of the public. We're also going to protect property and the
[06:45.290 --> 06:50.690]  environment. Just because we're in the process of putting a fire out doesn't necessarily mean
[06:50.690 --> 06:56.690]  we want to make things worse, right? So, for example, if we have a situation that might
[06:56.690 --> 07:01.390]  involve hazardous materials, we need to make sure that we're doing our best to prevent that from
[07:01.390 --> 07:08.210]  making the situation worse. And we want to minimize, excuse me, the disruption of community activities.
[07:08.930 --> 07:15.230]  Great example of that would be a car fire or a car accident. You have a lot of people who are,
[07:15.230 --> 07:19.170]  say, trying to get to work in the morning or trying to get home. And if one of those things occurs,
[07:19.170 --> 07:24.890]  we need to set up traffic support to help you still get from place to place and minimize any
[07:24.890 --> 07:31.450]  disruption while we're taking care of the particular incident. We, too, have a preparedness cycle.
[07:31.450 --> 07:35.830]  It looks kind of like this. We'll do some preparing. Then an emergency will happen.
[07:35.830 --> 07:41.550]  We'll respond to it. We're going to recover from it. We'll mitigate where we can. And we're
[07:41.550 --> 07:47.910]  back to preparation. Now, in this preparedness cycle in the fire service, we usually start more
[07:47.910 --> 07:55.430]  on the mitigation side. And let me show you how and why. So, mitigation for the fire service has
[07:55.430 --> 08:00.250]  to do with preventing future emergencies or minimizing their effects. So, if you've ever
[08:00.250 --> 08:06.610]  been to any kind of, say, fire department open house where they're doing fire safety demonstrations,
[08:06.610 --> 08:11.830]  they're handing out smoke detectors, right? That's a mitigation. We want to see if we can
[08:11.830 --> 08:17.370]  help people understand how to prevent fires and accidents in the first place. Sprinklers in a
[08:17.370 --> 08:23.150]  building are another great mitigation method. Preparedness. We need to be prepared to handle
[08:23.150 --> 08:27.490]  that emergency. So, we're going to make sure our gear is maintained, that we're drilling so we know
[08:27.490 --> 08:32.050]  how to use the equipment on the truck, that we have pre-plans of the building. Perhaps there's
[08:32.050 --> 08:38.370]  something unique about accessing certain parts of the building or getting people out if necessary.
[08:38.490 --> 08:43.210]  And then recovering. When we recover from an emergency, we're going to clean up. We're going
[08:43.210 --> 08:48.410]  to restore things to their functional purpose again. We're going to repair equipment that
[08:48.410 --> 08:55.670]  might have been damaged. This is pretty standard for us. So, here's the next part of the story.
[08:55.670 --> 09:01.550]  You saw me looking at the camera, right? This is the same fire. Two more pictures from this fire.
[09:01.910 --> 09:09.590]  The one on the left has me, if you probably can't tell because my name's not on my coat, but that is
[09:09.590 --> 09:16.470]  me. And what I'm attempting to do in that picture is change out an air bottle. The way firefighters
[09:16.470 --> 09:22.330]  breathe inside a building that is on fire is we wear something called self-contained breathing
[09:22.330 --> 09:32.090]  apparatus. And that's made out of a harness and a bottle and some other pieces that help redirect
[09:32.090 --> 09:39.430]  airflow. Needless to say, there are several components. My job on the left-hand side is to
[09:39.430 --> 09:46.570]  change the air bottle. Your average air bottle is about 30 minutes, 45, which can go very quickly if
[09:46.570 --> 09:52.730]  you're working hard in a scene. And when a firefighter runs out of air or starts to run out of air,
[09:52.730 --> 09:58.350]  they come out and they ask somebody to change that bottle. So, that was my job at this fire. I was
[09:58.350 --> 10:03.730]  changing bottles for the people who needed them. If you look in the picture on the right-hand side,
[10:03.730 --> 10:08.890]  you'll get at least a small idea of what that bottle and harness setup looks like, because you
[10:08.890 --> 10:13.810]  can kind of see the top of the bottle. What I want you to focus on in that right-hand picture
[10:14.630 --> 10:21.530]  is the fact that the firefighter's shoulder has a lot of damage. If you look carefully,
[10:21.530 --> 10:27.690]  you will see there's actually been flame and heat impingement to that shoulder area.
[10:28.350 --> 10:36.430]  What I didn't realize when I was changing that bottle is that it was not just his coat that had
[10:36.430 --> 10:43.790]  been damaged, but in fact the harness that holds the bottle and the air hoses that connect to it.
[10:44.470 --> 10:49.810]  So, I go to change that bottle, and unfortunately when I put the new bottle back in and connect it
[10:49.810 --> 10:57.030]  all up, I get hit with a blast of air in the face. Fortunately for me, I was wearing my gear correctly,
[10:57.030 --> 11:03.990]  there was no damage, and ultimately I instructed the firefighter to take the pack and make sure
[11:03.990 --> 11:11.250]  that it went out of service. Now, why is this important? Nobody got hurt, it wasn't a really
[11:11.250 --> 11:18.490]  huge deal, nothing bad ultimately happened. We'll get back to that. So,
[11:18.490 --> 11:23.750]  pay close attention to that picture, think about that story, and remember here, I was just doing
[11:23.750 --> 11:30.410]  my job. All right. So, one other piece of terminology in the fire system that's really
[11:30.410 --> 11:36.290]  important to understand is the incident command system. And it's this idea of an organized response
[11:36.290 --> 11:43.590]  to a problem. You're going to have a coordinated response with potentially multiple agencies,
[11:43.590 --> 11:51.030]  you need some way to organize that chaos, right? Such that everybody knows what they're doing and
[11:51.030 --> 11:56.430]  what their job's supposed to be. If you've been in the military before, you're probably familiar
[11:56.430 --> 12:01.230]  with chain of command. Obviously, if you're in law enforcement, that's something else you're
[12:01.230 --> 12:07.310]  familiar with. Not dissimilar idea. This is what forms our effective chain of command.
[12:08.790 --> 12:12.850]  So, here's what the incident command system looks like. And we're not going to spend time looking
[12:12.850 --> 12:17.470]  at all of this. But you'll see the incident commanders at the top, typically going to be
[12:17.470 --> 12:22.750]  one of your chief officers. And at the bottom, you're going to have operations and planning,
[12:22.750 --> 12:28.090]  logistics, finance. And in between, you'll have some other folks who report directly to the
[12:28.090 --> 12:33.290]  commander. What's important to note here is that the incident commander doesn't have a ton
[12:33.290 --> 12:38.750]  of people reporting directly to them. And the reason is, it makes this whole situation
[12:39.450 --> 12:46.130]  way more manageable. So, let's look at the pieces that are most often in smaller incidents.
[12:46.810 --> 12:52.450]  So, you have your incident commander. They're going to define what the goals and the objectives
[12:52.450 --> 12:56.670]  are for that particular incident. Again, usually your chief, or it might be a senior officer if
[12:56.670 --> 13:04.230]  no chief is available. And then you'll have operations. Operations is often the person who
[13:04.230 --> 13:10.850]  is in charge of making sure that those goals and objectives get met by figuring out what the
[13:10.850 --> 13:15.830]  strategy and the tactics are going to be. We sometimes have external operations and internal
[13:15.830 --> 13:21.190]  operations. If it's a large building, we might have operations on multiple floors. So, we might
[13:21.190 --> 13:28.650]  have operations for the first floor. We have a high-rise facility and sometimes we need
[13:28.650 --> 13:35.550]  multiple levels of operations. So, that's not uncommon. The last three here, logistics planning
[13:35.550 --> 13:42.110]  and admin finance, are more common in very, very large incidents. When you need to have lots of
[13:42.110 --> 13:47.010]  people and extra supplies and equipment brought in. If there's going to be a long-range plan,
[13:47.010 --> 13:52.670]  something like a hurricane hits and we have to mobilize troops or demobilize them quickly,
[13:52.670 --> 13:57.070]  that's going to be on the part of planning. And then if we need some kind of licenses or deal
[13:57.070 --> 14:02.450]  with compliance issues, get some money together, whatever, that's where admin and finance is going
[14:02.450 --> 14:08.790]  to come in play. All right. So, let's do a comparison of the two and kind of see,
[14:08.790 --> 14:16.770]  I think you'll find it interesting to see how they do compare. So, methodology-wise,
[14:17.010 --> 14:22.190]  I think you've seen they're very, very similar, right? The only thing that was really missing
[14:22.770 --> 14:28.270]  from the firefighting circle were the lessons learned. And I'll tell you, even though it's not
[14:28.270 --> 14:33.930]  officially part of the methodology, if you are good at what you do as a firefighter, you absolutely
[14:33.930 --> 14:39.110]  pay attention to lesson learned. So, I would say that these really, in terms of their functionality,
[14:39.110 --> 14:44.970]  are really very similar. But let's talk about some misconceptions.
[14:45.830 --> 14:50.890]  In incident response, there's this idea that every event is an incident. Well, that's certainly not
[14:50.890 --> 14:56.850]  the case. You can have an event where something is, you know, changed, but it's not necessarily
[14:56.850 --> 15:02.410]  an incident on the part of the particular organization you're with. And not every
[15:02.410 --> 15:08.370]  incident is handled the same way, right? I mean, certainly a small incident involving maybe
[15:09.010 --> 15:12.630]  one social security number is going to be different than an incident involving
[15:13.290 --> 15:19.290]  a hundred social security numbers or a thousand. Every incident is quickly solved. Well, we know
[15:19.290 --> 15:24.870]  that's not true. We can have an incident that's really fast, maybe gets resolved in 10 minutes,
[15:24.870 --> 15:29.930]  and it could take one that takes days, weeks, months. It can be a really long time, depending
[15:29.930 --> 15:34.090]  on how big the incident is. If law enforcement is involved, it can be even longer than that,
[15:34.090 --> 15:39.730]  depending on what court cases and all the other details play into it.
[15:40.310 --> 15:45.590]  This is, I think, one of the most important ones. There's this idea that every person on an incident
[15:45.590 --> 15:50.630]  response team needs to be a rock star. Well, that's not true. I'm not a rock star. I'll tell
[15:50.630 --> 15:56.250]  you that right now. And as new people come into this field, we shouldn't expect them to be rock
[15:56.250 --> 16:03.590]  stars either. We can all learn from each other. Another thing that's a misconception in IR is this
[16:03.590 --> 16:11.170]  idea that we can accurately attribute whatever's happened to a particular entity. It's hard enough
[16:11.170 --> 16:17.430]  to, you know, figure out attribution at all, but to do it accurately is even harder. Likewise,
[16:17.430 --> 16:22.810]  in firefighting, we have some misconceptions. How about this idea that firefighters are always paid?
[16:23.050 --> 16:27.510]  How many of you know that there are tons of firefighters around this country
[16:28.250 --> 16:34.270]  that aren't paid? I'm one of them. I've been a volunteer firefighter for many years.
[16:34.270 --> 16:41.450]  I've never, ever earned a dime from firefighting. And yet, I still love it. It's a fantastic hobby
[16:41.450 --> 16:46.810]  and one that I wouldn't trade for anything. There's also, from television especially, this
[16:46.810 --> 16:52.450]  idea that firefighters are big tough dudes. Well, I know you can only see my floating head, but guess
[16:52.450 --> 16:59.690]  what? I'm not a big tough dude. Never was, never will be. We also don't fight fires the same way
[16:59.690 --> 17:05.170]  every time. Again, yes, we put the wet stuff on the red stuff, as one of my fire instructors
[17:05.730 --> 17:13.430]  has said, but we don't do it the same way. Maybe we need to do a search and rescue first. Maybe
[17:13.430 --> 17:18.010]  the building has partially underground. There are all sorts of challenges with the situation
[17:18.010 --> 17:25.050]  that don't allow us to do the same exact thing every time. Certainly, there's this idea that,
[17:25.050 --> 17:29.770]  you know, fires get extinguished quickly, especially, again, on television shows,
[17:29.770 --> 17:36.630]  but it can take days. And in large, really large situations, it can take longer than that.
[17:36.770 --> 17:42.570]  Just depends. Maybe, maybe all you need is a water can and it's a little bit of burning mulch.
[17:42.570 --> 17:48.890]  That's extinguished quickly, but not all of them. And definitely not a full-time job,
[17:48.890 --> 17:54.070]  because that's not my full-time job. My full-time job is incident response.
[17:54.130 --> 18:01.030]  But we'll see, they're pretty similar things in some ways. First, we'll talk about the differences.
[18:02.070 --> 18:07.590]  So, in terms of incident response, it rarely involves life safety, except in the universe
[18:07.590 --> 18:13.090]  of healthcare. If you work in healthcare, you absolutely can be the difference between life and
[18:13.090 --> 18:18.630]  death in some of the incident response situations you might wind up in. Patient records are critical.
[18:19.210 --> 18:22.990]  Ideally, there are things in place to prevent that from being, you know,
[18:22.990 --> 18:29.590]  catastrophic. But in the rest of the world, despite what everyone would have you believe,
[18:29.590 --> 18:35.230]  life safety isn't usually the issue. That's important to remember.
[18:36.090 --> 18:41.310]  Certainly, IR teams typically get paid, and as I already mentioned, firefighters, not so much.
[18:41.790 --> 18:49.010]  We use computers and software for our day-to-day job. And while firefighters also sometimes use
[18:49.010 --> 18:55.170]  computers in certain capacity, they use mostly water, foam, chemicals, hoses, and show up in
[18:55.170 --> 19:00.090]  special vehicles, like fire trucks. Most of us doing IR, drive whatever we want, doesn't really
[19:00.090 --> 19:04.810]  matter. Obviously, if you're in an organization like law enforcement, well, you might have different
[19:04.810 --> 19:10.430]  experience. And we wear special gear as firefighters. I'm not going to show up at a fire
[19:10.430 --> 19:16.790]  in just a t-shirt and shorts. I'm certainly going to put on my gear, right? So those are some
[19:16.790 --> 19:24.210]  differences. Now, here's where they are similar. In both cases, we're going to focus on that
[19:24.210 --> 19:30.130]  immediate issue first and find that cause later. We need to put that fire out. We need to stop that
[19:30.130 --> 19:37.110]  attacker. We'll deal with attribution and the cause later. We're also going to use triage to
[19:37.110 --> 19:42.070]  determine the best course of action. If you're not familiar with triage, it's this idea that
[19:42.070 --> 19:47.470]  when you have more than one thing to consider, you're going to decide which of those things is
[19:47.470 --> 19:54.510]  most critical. So what most people are familiar with in terms of triage is a situation where
[19:54.510 --> 20:01.430]  there's been a motor vehicle accident and you have multiple people who are injured. You might
[20:01.430 --> 20:09.130]  have somebody who just has a bump on the head. You might have somebody who maybe is beyond saving.
[20:09.130 --> 20:14.210]  And you might have somebody who's serious, but with the right interventions could easily be
[20:14.210 --> 20:20.290]  saved. And looking at all those details, you have to decide which one to treat first.
[20:20.290 --> 20:25.710]  Fire situations are exactly the same thing. You're going to want to take a look at the facts
[20:25.710 --> 20:31.970]  and make some decisions about what to do first. Ideally, you've pre-planned that,
[20:31.970 --> 20:36.390]  but we'll come back to that. They're cyclic in nature, both of them. We've seen that already,
[20:36.390 --> 20:42.150]  right? And they both require some pretty interesting thinking outside the box.
[20:42.330 --> 20:46.930]  In incident response, sometimes we use tools that we might use for other things
[20:47.610 --> 20:55.590]  to try to get information for the response. And in firefighting, we do the same kinds of things
[20:55.590 --> 21:00.270]  in the sense that, for example, I have a friend who's a firefighter in my hometown
[21:01.430 --> 21:08.610]  who had a patient on a multi-story building who was very large and they didn't have equipment
[21:08.610 --> 21:12.970]  that would easily facilitate getting him out of the building where he was.
[21:12.970 --> 21:20.090]  So he went to the local hardware store, got some equipment, and built something to get this
[21:20.090 --> 21:25.190]  individual out of where he was stuck. Thought outside the box, didn't just use the equipment
[21:25.190 --> 21:32.250]  he had, worked with other things that he was able to get a hold of. In both cases, we often
[21:32.250 --> 21:36.830]  bring in outside entities. Firefighters rely heavily on mutual aid, bringing in other fire
[21:36.830 --> 21:42.630]  departments as necessary or perhaps specialty teams. We do the same thing in incident response.
[21:42.630 --> 21:47.990]  While I do incident response for the university, we might need another team if something were
[21:47.990 --> 21:52.970]  significantly serious enough to come in and help us. Maybe if it's a large enough incident,
[21:52.970 --> 21:59.010]  maybe if it's a certain kind of incident. And sometimes in both cases, there are inside teams.
[21:59.030 --> 22:04.450]  So again, I'm inside at the university and there are companies that have inside
[22:04.450 --> 22:08.870]  firefighting teams because they're a manufacturing organization that's large enough.
[22:10.010 --> 22:14.750]  So now let's talk about the things I've learned from being a firefighter that I think are relevant
[22:14.750 --> 22:21.550]  to IR specifically. I'll start with this first challenge, tunnel vision. This is an idea where
[22:21.550 --> 22:27.670]  we focus exclusively on one particular thing instead of seeing that big picture, right? And
[22:27.670 --> 22:34.890]  we make bad decisions when we do that. Think back to that image, the two images of me changing out
[22:34.890 --> 22:42.590]  that air bottle for the firefighter and his burn coat. This is an example of tunnel vision. I'm
[22:42.590 --> 22:49.550]  focused on doing my job. That's all I'm doing. And frankly, I could have gotten really hurt here
[22:49.550 --> 22:57.450]  because of tunnel vision. I wasn't taking a step back in my head and looking at the bigger picture.
[22:57.450 --> 23:03.510]  I was just focused on the task at hand. Here's another example from firefighting.
[23:03.510 --> 23:10.150]  Look carefully at those cones. Pull up to the fire, get off the fire truck, and this is what you see.
[23:10.650 --> 23:16.950]  What are those cones protecting? At first glance, you see a fire hose. No big deal. So maybe you're
[23:16.950 --> 23:22.210]  going to use that hose. You're going to help with the fire out. Maybe you're going to bring a piece
[23:22.210 --> 23:28.330]  of gear to the other person if you look carefully who's standing at the door. If you look more
[23:28.330 --> 23:35.550]  carefully, you may or may not see there is actually a live power line that those cones
[23:36.310 --> 23:42.130]  are actually trying to protect you from. And if you go rushing into the scene because you're so
[23:42.130 --> 23:48.690]  focused on putting the fire out or getting the gear that somebody needs, you could get really hurt.
[23:48.930 --> 23:54.850]  We have the same kind of thing in IR. We need to consider whether or not a scene is safe.
[23:54.850 --> 24:00.130]  In many cases, sure, it might be, but it depends. It depends on the situation.
[24:00.330 --> 24:07.270]  The picture that I'm showing you below is of a server room that has asbestos. If there's an issue
[24:07.270 --> 24:13.230]  in a server room with asbestos, you don't want to go running in there. Consider if somebody tells
[24:13.230 --> 24:19.190]  you that you have a malware situation and you get focused on that, but it turns out it's just
[24:19.370 --> 24:25.990]  a misconfiguration. Or maybe you do blind hardware acquisition, so you go running in and you grab
[24:25.990 --> 24:34.410]  drives, but that machine you find out later was powered on and was running encryption and now you
[24:34.410 --> 24:38.710]  don't have the keys anymore because all you've done is grabbed hardware. You haven't thought
[24:38.710 --> 24:45.970]  about it. Maybe you were told that a situation involved ransomware and you're worried now and
[24:45.970 --> 24:50.190]  you're focused on how you're going to handle the ransomware, but it turns out it's just a phishing
[24:50.190 --> 24:58.990]  email. So hand-in-hand with tunnel vision is reactionary behavior. Not only are we focused on
[24:58.990 --> 25:06.090]  one thing, but we react to it without thinking ahead, right? We're allowing those outside forces
[25:06.630 --> 25:12.510]  to make that decision instead of relying on the data we have at hand and thinking about that
[25:12.510 --> 25:21.390]  bigger picture. The fire service, it can lead to really deadly situations. So the photograph you
[25:21.390 --> 25:27.510]  see here is from a horrible fire that happened not all that long ago in Worcester, Massachusetts.
[25:28.210 --> 25:34.380]  It was a cold storage fire in an old meat packing plant that was built in 1906.
[25:35.290 --> 25:41.670]  People hadn't been in that building in a very, very long time. It had been vacant. No pre-plans
[25:41.670 --> 25:49.450]  had been made. It turns out it had a maze of meat lockers in it. When this fire broke out, there was
[25:49.450 --> 25:55.670]  supposedly somebody trapped inside. So they sent in two firefighters. The two firefighters went in
[25:55.670 --> 26:02.670]  and attempted to find the individual they thought was trapped. They got hopelessly lost and radioed
[26:02.670 --> 26:11.550]  for help. So the chief sent in two more firefighters to help them. Unfortunately, those two firefighters
[26:11.550 --> 26:15.830]  also got hopelessly lost and could not get out of the building, nor could they find the two
[26:15.830 --> 26:23.210]  original firefighters that had been sent in. So the fire chief sends in two more firefighters.
[26:23.910 --> 26:32.530]  Guess what? They too get hopelessly lost. And ultimately, the fire chief says, no more.
[26:32.670 --> 26:40.310]  I'm not sending any more in. Now, at first he was harshly criticized by his team for this,
[26:40.310 --> 26:46.650]  because, of course, their own folks were inside that building. But the reality is,
[26:46.650 --> 26:51.230]  any further reactionary behavior to try to save these individuals would just have led to more
[26:51.230 --> 26:58.970]  death. These folks died because pre-plans weren't made, and they didn't know what was in there,
[26:58.970 --> 27:02.470]  and it would have been a reaction to just keep sending firefighters in.
[27:03.090 --> 27:11.110]  We see the same thing within IR, although usually not as deadly, right? If we're doing forensics,
[27:11.110 --> 27:17.390]  we need clear, explicit goals. If the reactionary behavior is just figure out the bad stuff,
[27:17.390 --> 27:22.690]  that's not helpful. You need to know more specifically what it is they want you looking for.
[27:23.710 --> 27:28.210]  Another example is just pulling a network connection. What if you need more information
[27:28.210 --> 27:31.970]  about the attacker because of what they're doing on other systems?
[27:32.270 --> 27:37.450]  Or what happens if you just turn off a machine? We've already talked about the situation where,
[27:37.450 --> 27:44.370]  you know, you have a key in memory and you turn the machine off and it's gone. Suspending all
[27:44.370 --> 27:48.830]  accounts is another example where it could be you have one account that's compromised,
[27:48.830 --> 27:51.650]  now you suspend them all, and that causes more headaches.
[27:53.710 --> 28:01.310]  Another challenge is freelancing. Now, in our field in IT, a freelancer we often think of as
[28:01.310 --> 28:08.610]  somebody who's more of, perhaps, somebody who's just not working directly for the company, right?
[28:08.610 --> 28:13.630]  They're a contractor. But in the fire service, this notion of freelancing is this idea that
[28:13.630 --> 28:18.150]  somebody is not following chain of command, they go off on their own and kind of do what they want
[28:18.150 --> 28:24.170]  in an attempt to get the job done, but they're not following what's been laid out for them.
[28:24.730 --> 28:29.910]  And that can lead to dangerous and reckless situations and bad feelings and all kinds
[28:29.910 --> 28:37.110]  of problems. In the fire department, we see this typically where we have what's called the
[28:37.110 --> 28:41.690]  crossing of the streams. And for those of you who don't know, the picture in the bottom right
[28:41.690 --> 28:46.010]  is Ghostbusters, and they always talk about never cross the streams.
[28:46.610 --> 28:50.850]  Well, in firefighting, the idea is if I'm streaming water through the front of a building,
[28:50.850 --> 28:56.210]  and then a team comes to the rear of a building, and that building is a small house,
[28:56.210 --> 29:02.330]  now I'm fighting the team that's putting the water in, in both directions. And that could
[29:02.330 --> 29:08.330]  be really dangerous. Another way we see this freelancing in the fire department that can
[29:08.330 --> 29:13.790]  be really dangerous is something where doors are opened or windows are smashed
[29:14.330 --> 29:20.450]  without coordinating it with other people. We've seen situations where a window could get smashed
[29:20.450 --> 29:23.990]  and a firefighter could be standing right on the other side of it because they don't realize
[29:23.990 --> 29:29.710]  somebody's there. It can also ultimately feed the fire because where there's more oxygen,
[29:29.710 --> 29:34.010]  that fire is going to go. So if it hasn't been planned out and it's not coordinated,
[29:34.010 --> 29:36.930]  it can actually make things significantly worse.
[29:38.330 --> 29:44.390]  In incident response, we see the same kinds of things. A team goes in to do a response,
[29:44.390 --> 29:48.450]  and you get somebody who thinks they're just going to collect everything. Well, there could
[29:48.450 --> 29:54.410]  be legal ramifications for that. What happens if they don't collect what's absolutely required?
[29:54.430 --> 29:59.770]  Maybe they overlook something. Maybe there's a duplication of data collection which wastes time.
[29:59.770 --> 30:05.750]  Or maybe data gets altered or misrepresented because people are freelancing and not everybody's
[30:05.750 --> 30:15.910]  not following the same set of rules or the same guidance. Patience. This is a fantastic opportunity.
[30:17.010 --> 30:23.290]  Ultimately, you want to take that moment to determine what the best course of action is.
[30:23.290 --> 30:29.290]  As one of my fire instructors would say, slow is fast, right? You want to be deliberate about what
[30:29.290 --> 30:37.710]  you do because ultimately it will take less time if you take a moment, think things through,
[30:37.710 --> 30:46.210]  and then proceed. So in terms of firefighting, you may remember the same house here. It's the
[30:46.210 --> 30:52.090]  same story on the left is what we pulled up to. You'll notice you can see flame in the picture
[30:52.090 --> 30:58.150]  on the right hand side, the sort of orangey bit that's on the right part of that underneath the
[30:58.150 --> 31:04.190]  eaves next to the downspout. And you'll see a little bit of a glow toward the front of that
[31:04.190 --> 31:10.130]  third window on the right. But we don't see flame blowing through the roof. Now what I was seeing in
[31:10.130 --> 31:15.490]  that thermal camera image was that all of the fire, all of the significant flame and heat
[31:16.050 --> 31:23.970]  for this particular fire, that was up in that roof area. So what would have been ideal is for
[31:23.970 --> 31:33.510]  somebody to go in on the inside, break through from the second floor up into that attic area,
[31:33.510 --> 31:41.210]  and ultimately put the fire out. Unfortunately, we had people who focused solely on where they
[31:41.210 --> 31:48.210]  saw flame. And as a result, we see what we see on the right hand side where flame
[31:48.770 --> 31:55.910]  blows through the roof and there's a lot more of it. And it took us a lot longer to get this
[31:55.910 --> 32:01.210]  fire out. Which isn't to say that what everybody did was wrong, but taking a beat and thinking
[32:01.210 --> 32:08.490]  this through might have saved a lot of time with this fire. Okay, patience in terms of incident
[32:08.490 --> 32:16.170]  response. For those of you not familiar with Chegg, Chegg is a company that rents textbooks.
[32:16.170 --> 32:23.350]  Now what do you think the odds are of an entity that rents textbooks to students in higher ed,
[32:23.350 --> 32:30.550]  of them using, oh I don't know, same credentials at the university that they use at Chegg or vice
[32:30.550 --> 32:39.030]  versa? Yeah, pretty darn good. So Chegg had a massive breach that involved millions of accounts
[32:39.850 --> 32:46.830]  and because they provide services to multiple universities, as you might expect,
[32:46.830 --> 32:54.090]  many institutions wound up with this exact problem. Tons of compromised accounts. But
[32:54.510 --> 33:03.450]  is it really the university's problem in the sense that is it their breach? No.
[33:03.450 --> 33:10.290]  What it is, is password reuse. Chegg had the breach. Yes, the universities and institutions
[33:10.290 --> 33:14.830]  have to do something about making sure those passwords get reset, right? That's important.
[33:15.150 --> 33:19.470]  But it's not the same as having those passwords having been breached on the
[33:19.470 --> 33:25.990]  systems that are actually at the university. That is an incident. That's pretty serious.
[33:26.450 --> 33:32.510]  So before you panic when you hear about a huge breach, even if it could impact your institution,
[33:32.510 --> 33:39.750]  think for a moment. Take that beat about exactly what the implications are. What is the real risk
[33:39.750 --> 33:46.470]  here? And also consider that time is important, but as we saw with the picture of the asbestos
[33:46.470 --> 33:52.190]  room, life safety is more important. So taking that beat can also be life-saving in incident
[33:52.190 --> 33:59.710]  response depending on what you're walking into. All right, accountability. It's this idea,
[33:59.710 --> 34:04.430]  at least in the fire service, of knowing where all your people are. You want to know if they're
[34:04.430 --> 34:09.670]  in the trucks. You want to know who came with one truck. You want to know who's gone inside a
[34:09.670 --> 34:15.290]  building. You want to know who's getting rehabbed. But really, in incident response, we do the same
[34:15.290 --> 34:20.730]  kinds of things. We want to make sure that we prevent hazards and duplication of efforts. We
[34:20.730 --> 34:24.170]  don't want to step on each other's toes or have any sort of direct interference.
[34:26.070 --> 34:33.290]  So on the left are accountability tags that we use in firefighting. And for example, we'll use
[34:33.290 --> 34:39.210]  one of those tags we'll place on the truck so that everybody knows what truck you came in on.
[34:39.330 --> 34:47.650]  That way, the officer on that truck knows where their people are at all time. They might put
[34:47.650 --> 34:51.790]  another, if they go inside the building, they'll use a second tag and put it with the accountability
[34:51.790 --> 34:56.450]  person at the door so that they know where they are inside the building.
[34:56.690 --> 35:03.010]  For incident response, we can do the same kinds of things. Perhaps wipe card access. If you're
[35:03.010 --> 35:10.390]  going inside a facility where you work, we'll show you what room you're in. And maybe you need
[35:10.390 --> 35:15.410]  something more detailed, like an incident access log, depending on the situation. So we can do
[35:15.410 --> 35:23.630]  accountability with IR, and it's important. Okay, pre-planning. Pre-planning is absolutely
[35:23.630 --> 35:28.910]  critical. You want to know your environment. You want to know through your own documentation
[35:28.910 --> 35:36.330]  and planning, third-party documentation, and anything else that you can think of before it
[35:36.330 --> 35:43.930]  becomes a problem. And in the fire service, we're going to do that with what are called pre-plans.
[35:43.930 --> 35:48.910]  And in incident response, we're going to do it with things like tabletop and pre-mortem exercises,
[35:49.510 --> 35:55.950]  so that we have this idea of what we might run into before we run into it, and prevent those
[35:55.950 --> 36:05.270]  gotchas. So this is the kind of thing we do in firefighting. On the left is a map of a building,
[36:05.270 --> 36:11.310]  basically. I know it's a little difficult to see, but the idea is that it maps out for us where the
[36:11.310 --> 36:18.830]  hydrants are, where the electrical shutoff is, and it might tell us if there's any hazardous
[36:18.830 --> 36:23.790]  materials in the area. The thing on the right might tell us a little more about the building
[36:23.790 --> 36:32.250]  construction. This is more of a general pre-plan that's going to tell us about other hazards,
[36:32.250 --> 36:37.370]  other information, but it's the same concept, right? It tells us what we need to know about
[36:37.370 --> 36:42.890]  the facility we're going into to keep us safe, and those that we're trying to rescue,
[36:43.350 --> 36:47.410]  and perhaps the general public, depending on the particular situation.
[36:47.790 --> 36:54.230]  We can do the same thing in incident response. We can do tabletop exercises. If you're not already
[36:54.230 --> 37:00.470]  familiar with backdoors and breaches, you should be. The Black Hills folks, John Strand and his
[37:00.470 --> 37:06.630]  crew, have built this amazing card game, which uses a 20-sided die and this set of cards,
[37:06.630 --> 37:15.530]  and it can help you build an entire event from the ground up, and then you can do the investigation
[37:15.530 --> 37:21.410]  using those cards. It's really interesting, and I highly recommend it if you haven't already tried
[37:21.410 --> 37:30.410]  them. But it can walk you through scenarios, everything from something happening in a natural
[37:30.410 --> 37:37.050]  disaster type emergency to, you know, the intern who's done the wrong kind of thing and made an
[37:37.050 --> 37:42.890]  accidental mess. Obviously, documentation we've talked about, right? Document your servers,
[37:42.890 --> 37:47.150]  document your workstations, document where your data is and what your network is like,
[37:47.150 --> 37:53.130]  and the more information you have, the better off you'll be. And hopefully it goes without saying,
[37:53.130 --> 37:57.550]  to a certain degree, that this would also include your incident response plan and whatever team
[37:57.550 --> 38:06.990]  you have. So to summarize, we want you to start thinking like a firefighter. I think that applying
[38:06.990 --> 38:12.710]  these firefighter ideas to incident response has really made me a better incident responder,
[38:12.710 --> 38:19.550]  ultimately. You want to avoid tunnel vision. You want to do what's called a 360 of the problem,
[38:19.550 --> 38:24.950]  which in firefighting might literally mean a 360 around the building, which is what we try to do,
[38:24.950 --> 38:29.330]  or at least somebody does when we get there, whether it might be the operations person,
[38:29.330 --> 38:34.450]  it might be the incident commander. But at a minimum, we always pull past the building that's
[38:34.450 --> 38:40.090]  on fire, or potentially on fire, so that we can see at least three sides of the building.
[38:40.430 --> 38:45.910]  That gives us that bigger scope. We want to make sure we're acting, we're deliberate,
[38:45.910 --> 38:52.930]  we don't just react. We don't want to be misled by emotion. Follow whatever plan that you and
[38:52.930 --> 38:57.790]  your colleagues have come up with. Don't just freelance. It can be, you know, physically
[38:57.790 --> 39:04.810]  dangerous in certain circumstances, and it can just be plain frustrating in others.
[39:04.950 --> 39:10.350]  Have patience and take that beat. You never know what you might have missed if you didn't.
[39:10.350 --> 39:18.210]  Those 30 or 60 seconds could save tons of time. And even in the fire service,
[39:18.210 --> 39:23.110]  taking those few seconds ultimately could save a life, even if you think going faster
[39:23.110 --> 39:31.170]  might have done it. Location. Make sure you know where your people are. Document, document,
[39:31.170 --> 39:40.630]  document your location. Both of your equipment, your people, where your data is, and pre-plans
[39:40.630 --> 39:45.310]  they rock, right? Less chaos, way easier to respond if you already have a plan.
[39:45.310 --> 39:50.430]  And consider something like the incident command system, so that you're organizing your response.
[39:50.970 --> 39:55.770]  The thing about the incident command system that's so cool is that ultimately it is fantastic
[39:55.770 --> 40:03.370]  for scaling to larger incidents. As I mentioned, you can have only one person in charge of just
[40:03.510 --> 40:08.690]  a few people. And if you scale that out, it works really well in larger situations.
[40:08.790 --> 40:14.350]  So bringing in outside entities, this is so much easier because everybody's on the same page.
[40:16.050 --> 40:24.650]  So, some final thoughts. I know that many of you have never thought about incident response
[40:24.650 --> 40:33.330]  in this way. And that's okay. That's the idea. Think outside that box. Try some things you've
[40:33.330 --> 40:45.290]  never tried before. You may find that ultimately it makes your job maybe more challenging in the
[40:45.290 --> 40:53.810]  And the reality is great things never come from comfort zones. So, do it. Push that envelope.
[40:54.670 --> 41:00.490]  Hopefully you will find these tactics helpful. I want to thank you for coming to hear me speak.
[41:00.490 --> 41:05.590]  And I want to thank Wall of Sheep so much for having me. And I hope you have an absolutely
[41:05.590 --> 41:09.330]  wonderful DEF CON and a wonderful rest of your day.
